Image processing device, image processing method, production system, product manufacturing method, and storage medium

ABSTRACT

In order to provide an image processing device capable of supporting installation of an imaging unit, an image processing device includes: an image acquiring unit configured to acquire an image; a feature information extracting unit configured to extract feature information from the image; and a first generation unit configured to generate installation support feature information used for supporting installation of the imaging unit on the basis of the feature information extracted by the feature information extracting unit.

BACKGROUND Field

The present disclosure relates to an image processing device and thelike capable of supporting installation of an imaging unit.

Description of the Related Art

For the purpose of production, quality checking, transportation, and thelike of products, technologies for controlling various robot devicessuch as a gripping device, a robot arm, and other actuators andconveying devices are known. In addition, technologies for imaging atarget (hereinafter, referred to as a work piece) using an imagingdevice such as a camera, monitoring devices and an operation status byperforming image processing on recorded image data, measuring a positionof a work piece, and performing inspection are known as well.

By using the imaging device and the image processing device describedabove, measurement and inspection through visual observation, manualassembly, manual alignment, and the like become unnecessary. Inaddition, switching between control instructions and operationcorrection of a robot device can be performed, and thus a systemperforming more varied desired operations can be constructed.

In addition, by performing switching between processing details of imageprocessing, different processing can be performed on one piece of imagedata, and thus measurement and inspection through analysis of aplurality of different areas, processing of an image, and monitoring canbe performed. If visual functions of a production system and a conveyingsystem are realized using an imaging device such as a camera and animage processing device, it is important to install the camera at anappropriate position with an appropriate posture within the system andsecure a desired composition with respect to a work piece and a relativepositional relation with other devices.

In addition, there are cases in which, for example, in accordance withan increase in the amount of distribution and an increase in the amountof production of products, relating to the conveying system and theproduction system as described above, processes are not able to becompleted using only a system that has been initially constructed, and asystem of an equivalent specification needs to be replicated(additionally installed) inside the same production factory, anotherproduction factory, or the like. In such replicated systems, theprocessing speed, the accuracy, and the reliability of the conveyingsystem and the production system that have been replicated are requiredto be equal to or greater than those of the conveying system and theproduction system that have been initially built. Thus, complicatedadjustment is repeated until the processing speed, the accuracy, and thereliability that are required are obtained while repeating processes ofadjustment of a position and a posture of the camera, checking of anoperation of the image processing device, checking of an operation ofthe system, and the like any number of times.

For example, in Japanese Patent Laid-Open No. 2020-109915, adetermination unit determining a type of a subject within an image isincluded, and imaging parameters and compositions that are appropriatefor imaging the subject are recommended to a user by referring to ahistory of past imaging methods from a result of the determination ofthe subject. For this reason, a user can be assisted such that an imagehaving a similar composition can be smoothly captured by referring tocaptured images of professional cameramen and past captured images.

However, in Japanese Patent Laid-Open No. 2020-109915, although nextimaging parameters and the like can be recommended on a GUI by referringto a history of imaging parameters at the time of previously imagingsubjects and the like, it is difficult to apply such a technology, forexample, to a manufacturing system requiring image measurement and imageprocessing of high accuracy. In other words, it is difficult to applysuch a technology to a system in which installation of a camera andadjustment of a position and a posture need to be performed forreproducing an imaging position in units of sub pixels to severalpixels.

For example, a process of taking out a work piece such as an electroniccircuit of several centimeters from a tray by employing a robot oraligning a work piece to be aligned on a discharge device in a conveyingsystem may be considered. In the case, generally, the accuracy requiredfor an image measuring device is frequently to a few a micrometers toseveral millimeters.

In such a case, if a camera is not appropriately installed, largemeasurement error is included in an image measurement result, and thus,the robot cannot grip a work piece with high accuracy, and a palletizing(alignment) for a conveyer is not performed well. In addition, there isa likelihood of a work piece being dropped, simply gripping a work piecefailing, or a work piece colliding with a conveying device in thevicinity thereof or the like and damaged due to an unexpected grippingposture of the work piece in some cases.

For this reason, in such a conveying system or a production system,relative positional relations and postures of a camera and variousperipheral devices (a robot, a conveying device, a workbench, and thelike) are important, and complicated adjustment operations as describedabove are necessary.

In addition, not only in a replicated system but also in a system thathas been initially built, the camera may need to be replaced, forexample, due to a deviation of the position of the camera according tosome issue, a personal mistake, or the like or an occurrence of abreakdown of the camera during the operation of the system. Also in sucha case, there is an issue in that complicated adjustment operations asdescribed above are necessary.

There is a need in the art to provide an image processing device capableof solving the issues described above and supporting installation of animaging unit.

SUMMARY

According to one aspect of the present disclosure, in order to solve theissues described above, an image processing device includes at least oneprocessor or circuit configured to function as: an image acquiring unitconfigured to acquire an image; a feature information extracting unitconfigured to extract feature information from the image; and a firstgeneration unit configured to generate installation support featureinformation used for supporting installation of the imaging unit on thebasis of the feature information extracted by the feature informationextracting unit.

Further features of the present disclosure will become apparent from thefollowing description of embodiments with reference to the attacheddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an entire configuration diagram of a production systemincluding an image processing device 101 according to an embodiment.

FIG. 2 is a flowchart illustrating an example of a system operation setto a system control device 102 according to an embodiment in detail.

FIG. 3 is a block diagram of an image processing device 101 according toan embodiment.

FIG. 4 is a diagram illustrating a flowchart generation screen used forgenerating an image processing program 307 according to an embodiment.

FIG. 5 is a diagram illustrating a search pattern registration screenaccording to an embodiment.

FIG. 6 is a flowchart illustrating a sequence for generatinginstallation support information according to an embodiment.

FIG. 7 is a diagram illustrating feature information used for camerainstallation support according to an embodiment.

FIG. 8 is an entire configuration diagram of a replicated systemaccording to an embodiment.

FIG. 9 is a diagram illustrating a search pattern registration screen ofthe replicated system illustrated in FIG. 8.

FIG. 10 is a diagram illustrating a camera installation support screenaccording to an embodiment.

FIG. 11 is a flowchart illustrating a camera installation supportprocess according to an embodiment.

FIG. 12 is a checking screen for a position/phase measurement result ofa work piece when a test execution button 511 is clicked in FIG. 5.

FIG. 13 is a diagram illustrating a measurement result checking screenafter camera position adjustment according to an embodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, with reference to the accompanying drawings, favorable modeof the present disclosure will be described using Embodiments. In eachdiagram, the same reference signs are applied to the same members orelements, and duplicate description will be omitted or simplified.

FIG. 1 is an entire configuration diagram of a production systemincluding an image processing device 101 according to an embodiment.

A system control device 102 is illustrated, and the entire operation ofthe production system is controlled by transmitting instructions fromthe system control device 102 to the image processing device 101, aloading device 103, a robot 104, and a discharge device 105. Inaddition, the system control device 102 has a computer such as a CPUbuilt thereinto and has a memory, which is not illustrated, builtthereinto, and controls the entire production system by reading computerprograms from the memory described above.

The production system illustrated in FIG. 1 has a loading step ofloading a tray 109 on which work pieces 110 as a plurality of targetobjects are loaded using the loading device 103. In addition, a positionand a posture of the work piece 110 are measured using the imageprocessing device 101 and a camera 106 as a first imaging unit. In otherwords, the image processing device 101 acquires measurement featureinformation used for measuring a position and a posture of a work pieceand measures the position and the posture of the work piece by searchingfor the measurement feature information described above in an imageacquired by the imaging unit. In addition, the work piece 110 is grippedon the basis of a result of the measurement by the robot 104 as agripping unit, is taken out from the tray 109, and alignment in thedischarge device 105, a working process, and the like are performed onthe work piece 110, whereby a product is manufactured.

The image processing device 101 has a computer such as a CPU builtthereinto and has a memory, which is not illustrated, built thereintoand functions as an image processing unit that reads a computer programfrom the memory described above and sets or performs various kinds ofimage processing. The camera 106 functions as an image acquiring unitand acquires an image of a target object by performing capturing of animage. Although the camera 106 and image processing device 101 isdisposed within the system as separate bodies in FIG. 1, the camera 106and the image processing device 101 may be integrally configured.

Similarly, the image processing system is composed of the imageprocessing device 101 and the system control device 102, and the imageprocessing device 101 and the system control device 102 may beintegrally formed. Furthermore, although the image processing device 101and the system control device 102 may be loaded in the same casing asthat of the robot 104, the camera 106, or the like, they will bedescribed as separate bodies in this embodiment.

The robot 104 has a robot control device built thereinto, receivescontrol instructions from the outside, and controls operations ofgripping mechanisms such as shafts of the robot, robot hands (endeffectors), and the like. In this embodiment, although the robot isillustrated to have the configuration of an articulated robot, auniaxial drive device, a rotation stage, an orthogonal robot, or thelike may be used as a drive device. Furthermore, the robot may be asystem device configured by combining a robot hand (an end effector)including a gripping mechanism and a suction mechanism and the like.

A stand 107 is used for fixing the camera 106 and a camera positionadjusting stage 108. The camera position adjusting stage 108 can adjusta spatial position and a posture of the camera in accordance with asignal from the outside or manually by performing axial adjustment.

In this embodiment, the system control device 102 will be described asperforming control instructions for the entire system. However, theproduction system may be operated in accordance with a controlinstruction from an operation device disposed outside the productionsystem illustrated in FIG. 1 or may be configured to be automaticallyoperated on the basis of a time chart set in advance. In addition, onthe basis of a result of processing performed by the image processingdevice 101 and the like, another image processing may be performed byanother device of a later stage, or the robot may be controlled by theother device described above on the basis of a result of the imageprocessing.

The image processing device 101 and the camera 106 are configured to beable to perform imaging in an arbitrary sequence using imagingparameters, process a captured image, and output a result thereof Thecamera 106 may include mechanisms used for changing various imagingparameters such as a pan, a tilt, a zoom magnification, a focaldistance, a diaphragm, a signal amplification rate, and the like. Inaddition, in accordance with an instruction from the image processingdevice 101 or the system control device 102, the camera is able tocontrol an imaging angle by controlling the pan and the tilt describedabove and change various imaging parameters such as the zoommagnification, the focal distance, the diaphragm, the signalamplification rate, and the like.

In a production system in which there are many repetitive operations asin FIG. 1 for an industrial use or the like, the image processing deviceis used as a visual sense of the robot and the conveying device, andthus, the operation of the entire production system is set in advance bythe system control device 102. In addition, a timing, parameters, acomposition, and the like of imaging that are preferable for imageprocessing are set in advance as well.

FIG. 2 is a flowchart illustrating an example of a system operation setto the system control device 102 according to an embodiment in detail.The system operation according to this embodiment is operated inaccordance with a computer program of a flowchart generated by a userusing the system control device 102.

However, after a system control operation is generated by a systemcontrol program generating device not illustrated in FIG. 1, the programmay be downloaded into a memory of the system control device 102.Alternatively, a package function including a computer program ofcombinations of flowcharts that are prepared typically for each functionor for each purpose in advance may be provided in the form of beingselected by a user on a GUI, parameters being adjusted, or the like.

In the flowchart illustrated in FIG. 2, the system control device 102loads the tray 109 on which a work piece 110 is loaded by the loadingdevice 103 and measures a position and a posture of the work piece 110using the image processing device 101 and the camera 106. The flowchartillustrates an example of the flow of the process of extracting the workpiece 110 from the tray 109 while gripped by the robot 104 and aligningthe work piece in the discharge device 105 in detail.

The system control device 102 starts a system operation in Step S201. Inaddition, in accordance with transmission of a control instruction fromthe system control device 102 to the loading device 103, the loadingdevice 103 moves a stage by a predetermined amount of feeding in StepS202. In accordance with this, the tray 109 on which the work piece 110is loaded is carried in to a predetermined position.

The system control device 102 transmits an image processing instructionto the image processing device 101 in Step S203. The image processingdevice 101 performs image processing using an image of a target objectacquired by the camera 106 and measures a position and a posture of thework piece 110. Then, when the position and the posture of the workpiece 110 can be determined by the image processing device 101, theimage processing device 101 is caused to reply to the system controldevice 102 with a result thereof At that time, the image processingdevice 101 may further measure a type of the work piece 110 using theimage obtained by the camera 106 and change the discharge position orperform sorting as necessary. Alternatively, a quality inspection ofwhether there is a problem of quality such as a defect in the work piece110 may be performed using image processing.

In addition, when the work piece 110 cannot be detected, the systemcontrol device 102 may determine that the amount of feeding of theloading device is insufficient and feed the stage of the loading device103 further. In addition, a status of the work piece 110 or the tray 109may be determined in accordance with a result of the process of StepS203, and, for example, when work pieces overlap each other or overflowfrom the tray, the system control device 102 may change a control speedof the robot 104 or stop the operation of the system.

In Step S204, in accordance with the replied information of the positionand the posture of the work piece 110, the system control device 102performs a correction operation of moving the robot 104 to the upperside of the work piece 110 or rotating a hand (an end effector).

The system control device 102 moves the hand of the robot 104 to aposition at which the work piece can be gripped by the hand (the endeffector), for example, to the right upper side of the work piece 110 orthe like in Step S205.

The system control device 102 causes the work piece 110 to be gripped(picked) by controlling the opening/closing of the hand in Step S206. Atthis time, for example, one work piece is gripped each time in FIG. 1.Then, the work piece is moved to the upper side of the discharge device105 using the robot 104 in Step S207. Furthermore, the system controldevice 102 moves the robot 104 to immediately above a predeterminedsetting position in correspondence with determination of the number oftimes of repetition in Step S211 of the flowchart and a loading quantityobtained from the outside in Step S208.

The system control device 102 sets (places) the work piece 110 to apredetermined setting position by controlling the opening/closing thehand in Step S209. Then, the system control device 102 causes the robotto move (withdraw) to the upper side of a carrier stand in Step S210.

When the number of operations of the robot is smaller than apredetermined number of times N1 (in the case illustrated in FIG. 1,N1=2), and the system control device 102 determines that setting of allthe two work pieces 110 corresponding to one row that is horizontallyaligned has not been completed in Step S211, the process returns to StepS203. Then, the operations of Steps S203 to S211 are performed again. Onthe other hand, when the number of operations reaches the predeterminednumber of times N1, the system control device 102 determines thatsetting of work pieces corresponding to one row has been completed, andthe process proceeds to Step S212.

In Step S212, when the number of operations is smaller than apredetermined number of times N2 (in the case illustrated in FIG. 1,N2=3), and it is determined that setting of all the work pieces 110corresponding to three rows on the tray has not been completed (No), theprocess returns to Step S202, and the loading device 103 advances thetray by one row. Then, Steps S202 to S212 are performed.

When the number of operations has reached the predetermined number oftimes N2 (in the case illustrated in FIG. 1, N2=3), and it is determinedthat setting of all the work pieces 110 corresponding to three rows hasbeen completed (Yes) in Step S212, the process proceeds to Step S213. Inaccordance with this, gripping of work pieces on one tray and a settingoperation for the work pieces are completed. The entire flow illustratedin FIG. 2 is performed again for the next tray.

In addition, in the case of such a production system, the imaging deviceand the image processing device can be not only used as the visual senseof the robot as described above but also used for the purpose ofmonitoring the operation status of the system. For example, continualmonitoring can be performed regarding whether work pieces loaded intothe loading device 103 or the tray 109 have collapsed, whether the robot104 has performed movement deviating from a desired operation, orwhether there is a situation hindering an operation near an operationarea.

FIG. 3 is a block diagram of the image processing device 101 accordingto the embodiment. The image processing device 101 includes aninput/output display device 301 as a display unit configured using aliquid crystal panel and the like and an operation input device 302 forvarious operation inputs that is configured using a keyboard and amouse, a touch panel, an input operation controller, a gesture inputdevice, and the like. The input/output display device 301 and theoperation input device 302 mainly configure a user interface.

In addition, the image processing device 101 may be connected to notonly the camera 106 but also the camera position adjusting stage 108 andthe like for control. Furthermore, a lighting device composed of ahalogen lamp, a light emitting diode lamp, or the like may beadditionally disposed for imaging, and an external storage device may beconnected for increasing a storage area. The units described above areconnected through an interface 303 disposed on internal buses of theimage processing device 101. The interfaces 303 are configured on thebasis of specifications that are appropriate for communicating with theunits described above. For example, the interfaces are composed of anetwork interface, a serial communication interface, and the like.

The image processing device 101 includes a CPU as a computer composed ofa general-purpose microprocessor and an arithmetic operation unit 304composed of an image processing processor and the like as controldevices that are processing bodies of image processing. The arithmeticoperation unit 304 is connected to a storage unit 305 through internalbuses (a data bus, an address bus, and other control lines, and thelike). The storage unit 305, for example, is composed of a ROM, a RAM,or a nonvolatile memory device such as an E(E)PROM. Alternatively, thestorage unit 305 may be configured using an external storage device (anHDD or a storage device composed of semiconductor elements which is notillustrated, an external storage device that can be connected to theinterfaces 303, and the like) and the like.

A data saving area 306 of the storage unit 305 is composed or a RAM areain the storage unit 305 or a file area, a virtual storage area, and thelike of an external storage device. The data saving area 306 is used fortemporarily storing processing data and is used for storing settingparameters and the like of image processing. In addition, an imageprocessing program 307 as a computer program for performing imageprocessing according to this embodiment is stored in the storage unit305.

The image processing program 307 changes settings and the like of imageprocessing and performs image processing in accordance with variousoperations performed using the operation input device 302 and the like.In addition, the image processing program 307 can save changed detailsin the data saving area 306 or delete the changed details. Furthermore,the image processing program 307 can transmit/receive data to/from anexternal control device through the interfaces 303, can be connected toan external storage unit and the like and store data in the externalstorage unit, and receive data from the external storage unit as aninput.

The image processing program 307, for example, is composed of softwarethat realizes the following functions. First, image processing 308 is amain part of the image processing program that realizes image processingdescribed below. An image processing library 309 is used in the imageprocessing 308. The image processing library 309, for example, is alibrary that is linked statically or dynamically and is mounted in thestorage unit 305. Image processing setting 310 that determines thebehavior of the image processing 308 is set in accordance with variousoperations performed by the operation input device 302 and the like.

In addition, an input/output (I/O) routine that realizes the followingfunctions is included in the image processing program 307. In otherwords, the I/O routine includes external device control 311, saving datageneration 312, an accept instruction 313 from an external controldevice or the like, and a temporary storage process 314 for temporarilystoring data in a RAM area, a cache area of the arithmetic operationunit 304, and the like. In addition, the I/O routine includes displayscreen generation 315, saved data output 316, an accept operation 317from the operation input device 302, and the like. Each of the functionsdescribed above is mounted in the storage unit 305 in the form of anapplication (utility) program or a subroutine composed of libraries thatare linked statically or dynamically.

By executing the image processing program 307, the image processingdevice 101, for example, can control the camera 106 and perform imageprocessing using the arithmetic operation unit of the image processingdevice. In addition, the image processing device 101 can receive a useroperation from the operation input device 302 or receive an instructionfrom an external control device or the like.

In accordance with such an operation or an instruction, the arithmeticoperation unit 304 can call each function of the image processingprogram 307 or the library, perform an arithmetic operation process, andtransmit a result of the image processing to the system control device102. In addition, the arithmetic operation unit 304 may accumulate (log)the result of the image processing in an external storage device.Furthermore, the arithmetic operation unit 304 can compose a screenconfiguration stored in the program in advance and the result of theimage processing as a screen and display the composed screen in theinput/output display device 301.

FIG. 4 is a diagram illustrating a flowchart generation screen used forgenerating the image processing program 307 according to the embodiment.By executing the flowchart generated here, the image processing device101 realizes Step S203 for performing image processing in the systemcontrol illustrated in FIG. 2. In addition, the screen illustrated inFIG. 4 can be displayed in the input/output display device 301. Theimage processing program 307 according to this embodiment operates inaccordance with the flowchart generated by a user using the imageprocessing device 101.

As another embodiment, a form in which, after the image processingprogram 307 is generated by an image program generating device notillustrated in FIG. 1, the image processing program 307 is copied to thestorage unit 305 of the image processing device 101 may be employed.Alternatively, the image processing program may be provided in a form inwhich a user selects a package function including combinations of imageprocessing flowcharts that are typically prepared for each function orfor each purpose in advance on a GUI or adjusts parameters.

In FIG. 4, a list 401 of processing parts of a flowchart is illustrated.A user performs drag & drop of parts from the list 401 to a flowchartarea 402 using a mouse of the operation input device 302 and joins theparts using lines, thereby generating a flowchart.

One example of the image processing flowchart 403 is illustrated. Here,an internal process of the image processing 203 in the operation flow ofthe system control will be described as an example.

The image processing device 101 accepts an image processing request fromthe system control device 102 in Step S404. At this time, a flowchart ofthe image processing may be designated, and image processing parametersfor execution and a time of used image data may be designated.

In the image data acquisition process of Step S405, the image processingdevice 101 acquires image data of a work piece as a target object imagedby the camera 106. Other than that, an image for processing may beacquired from recording data accumulated in the camera. For example, thedata to be acquired may be latest image data in the accumulatedrecording data or image data of a measurement time designated from thesystem control device.

The generated flowchart inspects (searches) present/absence of a workpiece in the acquired image data in Step S406. A presence/absenceinspection method, for example, may be acquired from the imageprocessing library 309 and may be a method in which a work piece isextracted, for example, using luminance information and colorinformation, and then presence/absence is determined according towhether an area of the work piece is equal to or larger than apredetermined value. Alternatively, the presence/absence may becalculated through pattern matching using shape feature information of adensity distribution and a luminance gradient, and pixels of whichluminance gradients are equal to or larger than a predetermined gradientmay be extracted as edge information, and the presence/absence may beinspected using a position, the center of gravity, an inclination, andthe like of the edge information.

It is determined whether a work piece is present/absent in Step S407. Ifit is determined that a work piece is not present (No) in accordancewith the inspection result obtained in Step S406, the result is recordedin Step S409, and a reply to the system control device 102 with theresult is performed in Step S410. If it is determined that a work pieceis present (Yes) in Step S407, the process proceeds to Step S408.

A position and a phase of the work piece are measured in Step S408. As amethod for measuring a position and a phase of a work piece, a method inwhich measurement feature information of local pattern image of adensity distribution of the work piece and the like, which is recordedand saved on a recording medium in advance, is read and obtained, and asearch for a position and a phase having a strong correlationcoefficient for the measurement feature information is performed from aninput image may be used. Alternatively, a unit configured to extractfeature information such as a shape of a luminance gradient of a workpiece and the like from an image and generate measurement featureinformation for measuring a position and a posture of a target object onthe basis of the shape feature information may be installed. Then, asearch for a position and a phase having a string correlationcoefficient on the input image may be performed using the measurementfeature information.

In Step S409 for recording results, data such as execution results anddetermination results acquired in Steps S404 to S408 and the like isrecorded. The start time in Step S404 for accepting a processing requestand details of certain execution options may be stored as well. Inaddition, image data obtained in Step S405 for acquiring an image, animage capturing time and an image size, imaging parameters, camerainformation, and the like may be stored as metadata. Furthermore,extracted area information of a work piece obtained in the inspection ofStep S406, density information and gradient information used for theinspection, color information, and the like may be stored. As a resultof execution of Step S406, a detected position and a detected phase of awork piece, a correlation value, and the like may be recorded.

An execution result of the flowchart generated in this way istemporarily stored through the temporary storage process 314 performedby the image processing device 101 or is stored in the data saving area306. In addition, the execution result may be referred to or deleted asnecessary, may be transmitted or received through the interfaces 303, ormay be stored or input through connection to an external storage unit orthe like. The results of Step S409 or some thereof are output in StepS410, and the flowchart ends. When the results from Step S409 areoutput, the results may be transmitted to the system control device 102,be stored in the data saving area 306 of the image processing device, orbe output to and displayed in the input/output display device 301.

In addition, when a user clicks on a button 411, the image processingdevice 101 can generate the image processing program 307 for executingthe flowchart displayed in the flowchart area 402. Furthermore, byinputting a new flowchart name to a list box 412, a user can add a newflowchart to this system. In addition, a user also can select aflowchart to be displayed in the flowchart area 402 using the list box412. When a button 413 is clicked, the generation of the imageprocessing program ends, and the flowchart area 402 is closed.

In addition, when a user double-clicks on any one of steps that areparts of the image processing flowchart 403, a transition to a settingscreen used for setting a detailed process of the part can be performed.Here, a method for registering feature patterns for realizing aposition/phase measurement function will be described with reference toFIG. 5 using a setting screen of Step S408 used for measuring a positionand a phase as an example.

In addition, a method for generating support data used for supportingthe installation operation if a user installs an initial (first) systemand then installs an equivalent system for the second time or subsequenttimes or if a user installs an initial system and then changes theinstallation state of the camera, or the like will be described.

FIG. 5 is a diagram illustrating a search pattern registration screenaccording to an embodiment, and a user interface for performing settingof image processing in Step S408 that is a position/phase measurementstep (part) for a work piece will be described with reference to FIG. 5.

In the position/phase measurement step S408, a search for a position anda phase having a strong correlation coefficient on an input image isperformed, or shape feature information of a luminance gradient of awork piece is extracted using a local pattern image of a densitydistribution of the work piece as a target object and the like. By usingsuch feature information, a search for a position and a phase having astrong correlation coefficient on the input image is performed. A GUIused for registering a pattern image (measurement feature information)for a search will be described with reference to the screen illustratedin FIG. 5.

A button 502 of a pull-down menu for selecting an input candidate of a“setting image” (an image that becomes a reference for setting imageprocessing) used for setting is illustrated in FIG. 5. A camera 106 isselected as a “used camera” in FIG. 5, and thus a “camera image” isdisplayed as the “setting image”, and, when a user clicks on the button502, a captured image acquired by the camera 106 is displayed as the“camera image”. Other than that, file names of images that have beensaved in the data saving area 306 of the image processing device 101 inadvance may be displayed as options in the form of a list, and a usermay be allowed to select one of the file names. When a user selects acharacter string from this list, the selected image is registered as asetting image.

The “camera 106” is selected as the “used camera”, and the “cameraimage” is selected as the “setting image” in FIG. 5, a captured imageacquired by the camera 106 is used as a setting image. When a button 502is clicked, an image acquisition step for acquiring an image isperformed, and the captured image (camera image) acquired by the camera106 is displayed in an image display area 503 as a setting image.

Next, the user sets a search pattern setting area 504 that is an areafor setting a search pattern (measurement feature information). For thesetting, first, an area shape is selected using a button 505.

As an area shape, a shape such as a rectangle, a circle, an oval, or aring can be selected from the pull-down menu using the button 505. Inaddition, area addition, area exclusion, and the like using acombination of such shapes may be performed. Furthermore, a free shapemay be set using the operation input device 302 such as a mouse. In thisembodiment, a case in which a rectangular search pattern setting area504 is set will be described as an example.

As setting items 506 for a rectangular area, upper left x coordinate/ycoordinate and lower right x coordinate/y coordinate of the rectangleare set. Here, an image coordinate system having the upper left side ofa captured image as an origin and having an x axis in the horizontaldirection, and having a y axis in the vertical direction is used. As amethod for designating the setting items 506 for the area, in additionto a method of directly inputting numerical values, a function ofproviding count-up/count-down buttons and adjusting the setting items byclicking these or the like may be installed.

At this time, the search pattern setting area 504 is displayed withsuperimposed on the image display area 503 and is updated and displayedat any time. In accordance with this, a user can set the search patternsetting area 504 while checking a setting image in the image displayarea 503. Here, the setting step described above using the GUIillustrated in FIG. 5 functions as a second generation unit (a secondgeneration step) for generating measurement feature information used formeasuring a position and a posture of a target object on the basis ofthe extracted feature information described above.

Next, a sequence for setting a search method using the search patternsetting area 504 will be described.

Here, for example, “shape feature” is selected as a search method usinga search method 507. The search method based on “shape feature” is amethod in which shape feature information of a luminance gradient of awork piece is extracted, and a search for a position and a phase havinga strong correlation coefficient for the search pattern setting area 504on an input image is performed using the feature information.

As the search method, other than that, at least one of “normalizedcorrelation”, “matching”, and the like can be selected. Here, the“normalized correlation” is a method in which local pattern informationand the like of a density distribution of a work piece are normalized,and a search for a position and a phase having a strong correlationcoefficient for the search pattern setting area 504 is performed. Inaddition, the “matching” is, for example, a method in which athree-dimensional shape is extracted as a model if the camera 106 canacquire three-dimensional information of a work piece, and a search forthe three-dimensional shape is performed inside a three-dimensionalspace, and a mutual distance is minimized. In addition, at least one ofsearch methods such as “shape feature”, the “normalized correlation”,“matching”, and the like may be configured to be selectable on the basisof a type of model of a camera as an imaging unit.

In addition, in the example illustrated in FIG. 5, “shape feature” isselected as the search method, and it is determined that the shapefeature has been able to be detected if a correlation value between thesearch pattern setting area 504 and a work piece exceeds a predeterminedthreshold. For example, if a correlation value exceeds a numerical valueset in a detection threshold 508 designated in advance, it is determinedthat the shape feature has been able to be detected.

Other than that, a search for a work piece may be performed for theinside the image with the search pattern setting area 504 beingtransformable to a specific enlargement/reduction range 509 or aperspective distortion (swing and tilt distortion) range 510, and a workpiece entering the range of a threshold may be detected. In the exampleillustrated in FIG. 5, the enlargement/reduction range 509 is set to 0.9times to 1.1 times, and thus a similar work piece can be detected fromthe inside of the image with a size difference of less than 10 percentfrom a set search pattern allowed.

In accordance with this, even if a distance from a work piece isundefined or there is an individual difference in the work piece, robustdetection of the work piece can be performed. In addition, since theperspective distortion range 510 is set to the range of ±5°, a similarwork piece can be detected from the inside of the image with a swing anda tilt of ±5° relative to the set search pattern (rotation around ahorizontal axis, rotation around a vertical axis, rotation around theimage, or the like) allowed. In accordance with this, even if a workpiece is tilted, robust detection of a work piece can be performed.

After the setting using 505 to 510 ends, when a test execution button511 is clicked, imaging is performed using the camera 106 selected asthe “used camera”, and measurement of a position and a phase of a workpiece is performed for an acquired image, and a result of themeasurement of the position and the phase of the work piece can bechecked.

FIG. 12 is a checking screen for a position/phase measurement result ofa work piece when the test execution button 511 is clicked in FIG. 5. Anexample in which similar feature information is detected in an image ofthe image display area 503 using a search pattern set in the searchpattern setting area 504, and a result thereof is displayed isillustrated.

Work pieces 1201 to 1206 from which shape features similar to those ofthe search pattern have been detected are highlighted and displayed.Here, although IDs are assigned in order of highest to lowest similarity(correlation), the IDs may be assigned in ascending or descending orderof detected coordinates, or IDs may be assigned on the basis of detectedsequences or the like. The number of work pieces 1207 from which shapefeatures similar to those of the search pattern have been detected isillustrated.

A window 1208 is used for selecting an ID assigned to each detected workpiece and is used for displaying detailed detection information for theselected ID. A window 1209 displays detailed detection information aboutthe ID selected in the window 1208. As the ID of the detected work piece1201, “1” is selected in the window 1208 in FIG. 12, and thus a resultindicating that the search has been performed on the basis of the “shapefeatures” as the search method, and a correlation value as a score is0.99, and differences in the position and the rotation phase are almost0 is displayed.

When a cancel button 513 is clicked in this state, the settings set asdescribed above and the setting change parts are discarded.

On the other hand, when an OK button 512 is clicked, the setting of thesearch pattern setting area 504 set as described above, the settings ofthe button 505 to the perspective distortion range 510 relating to theposition/phase measurement, and the detection results and the likeillustrated in FIG. 12 are saved in the data saving area 306 inassociation with the image processing program 307 and the like. Suchinformation is read when the program is executed or when a transition tothe image processing setting screen is performed.

In addition, if a camera installation support information saving button514 is checked when the OK button 512 is clicked, the process ofgenerating installation support information for supporting installationof a camera is performed, and a result of the process is also saved inassociation with the image processing program 307. This will bedescribed below.

FIG. 6 is a flowchart illustrating a sequence for generating camerainstallation support information (support information for installation)according to an embodiment. The sequence illustrated in FIG. 6illustrates a flow performed by the arithmetic operation unit 304 of theimage processing device 101 if the OK button 512 is clicked in a statein which a camera installation support information saving button 514illustrated in FIG. 5 is checked.

In addition, the flow illustrated in FIG. 6 may be performed when acamera image is input or may be performed in the background duringadjustment of the search pattern setting area 504. By configuring assuch, pre-processing such as outline extraction, shade calculation, andnormalization can be performed in parallel, a user's operation standbytime is reduced, and camera installation support information can begenerated smoothly.

The arithmetic operation unit 304 of the image processing device 101starts the flow in Step S601 and extracts peripheral features in StepS602. For a peripheral area of the search pattern setting area 504 inwhich the search pattern is set in a camera input image, featureinformation corresponding to shape features are extracted in Step S602.The reason for extracting feature information from the peripheral areais that there is a high possibility of the accuracy being higher iffeatures of not only work pieces but also peripheral environmentsthereof are used as information for performing installation of thecamera.

As a range of the peripheral area, if there are rooms in the capabilityand the time of the arithmetic operation unit 304, feature informationmay be extracted from an area including the search pattern setting area504, for example, the entire image display area 503. In addition,feature information may be extracted within a range of pixelscorresponding to a number set in advance.

In other words, feature information used for generating installationsupport feature information is extracted from a first area of the image,and feature information used for generating measurement featureinformation is extracted from a second area different from the firstarea in the image. In addition, it is preferable that the first area belarger than the second area.

Furthermore, feature quantities may be extracted from a peripheral areanot including the search pattern setting area 504. Here, Step S602, forexample, functions as a feature information extracting unit (a featureinformation extracting step) that extracts feature information of atarget object or the periphery thereof from an image.

In addition, if a feature is found within a predetermined range of theperiphery, the area may be widened while widening the feature extractionarea until a feature is found on the periphery.

FIG. 7 is a diagram illustrating feature information used for camerainstallation support according to an embodiment, and an example of shapefeatures (feature information) 701 extracted from the entire image inwhich a search pattern is set is illustrated.

Next, the extracted features 701 are separated in Step S603 illustratedin FIG. 6. A separation method may be considered to be based on variouskinds of information such as a position and a shape, a gradientintensity, a shade average, and a shade deviation, and such options maybe configured to be selectable in a GUI or a setting file. Here, forsimplification of description, separation is assumed to be performed onthe basis of connectivity (continuity) of the extracted features 701.For example, a column of “feature information” 702 is a result ofseparation of the features 701 based on the connectivity.

The separated features 701 are evaluated in Step S604. As an evaluationindex, information contributing when the position of the camera, inother words, a relative position between the camera and s work piece isadjusted is used. For example, the evaluation may be considered to bebased on various kinds of information such as a size of a feature shape,a magnitude of a shade deviation, distance information from the searchpattern setting area 504 on an image for which a model of work pieces isgenerated, and selection of such various kinds of information may beconfigured to be performed in a GUI or a setting file. Here, Step S604functions as an evaluation unit that evaluates feature informationextracted by the extraction unit.

Here, an evaluation rank (a priority level) is assumed to be determinedon the basis of a size on the image of the separated features 701 as theinformation described above. In other words, sizes of the separatedfeatures 701 are evaluated in Step S604, then sorting of the features isperformed on the basis of the sizes in Step S605, and a sequence that isprioritized to be used for camera adjustment support is determined. Inaccordance therewith, installation support feature information to whichpriority levels are assigned in accordance with evaluation is generated.

Here, Step S605 functions as a first generation unit (a first generationstep) that generates installation support feature information used forsupporting installation of the imaging unit on the basis of theextracted feature information.

An image diagram 702 of installation support feature informationgenerated by separating the feature information on the basis of theconnectivity and performing sorting using size information as anevaluation index is illustrated. IDs are sorted in order of prioritylevels in the image diagram 702.

Step S606 is a process of saving the generated installation supportfeature information. In other words, search information such as thesetting of the search pattern setting area 504 set as described aboveand settings of the button 505 to the perspective distortion range 510relating to measurement of a position and a phase is saved. In addition,by clicking the OK button 512, search information such as theinstallation support information that have been extracted and calculatedin Steps S602 to S605 is also saved in the data saving area 306 and thelike of the image processing device 101 in association with the imageprocessing program 307. Thereafter, the flow ends in Step S607.

In addition, the installation support information described aboveincludes the feature information represented in the features 701 and thefeature information 702, a size, a position, a phase, intensity gradientinformation, color information, an evaluation value, a priority level,which is used for camera installation support, an original image, whichis displayed in the image display area 503, and the like of each pieceof feature information. In addition, the installation supportinformation is used as search information for searching a target object.

Furthermore, the setting information and the installation supportinformation as described above may be transmitted/received to/from theoutside through the interfaces 303 together with being stored in thetemporary storage process 314 and the data saving area 306. In addition,the information described above may be stored in a connected externalstorage device or the like or may be input from an external storagedevice.

Next, a method for replicating a system equivalent to the productionsystem illustrated in FIG. 1 using the installation support informationand a captured image acquired by a camera, and image processing andsupporting installation of second and subsequent cameras in the sameproduction system will be described with reference to FIGS. 8 to 13. Inaddition, support is performed by evaluating reproducibility of aninstallation position and a posture of a camera, outputting theevaluation results, and further outputting equivalent image processingresults.

FIG. 8 is an entire configuration diagram of a replicated systemaccording to an embodiment and illustrates a replicated system of theproduction system illustrated in FIG. 1.

An image processing device 801 to work pieces 810 illustrated in FIG. 8have configurations similar to the image processing device 101 to thework pieces 110 illustrated in FIG. 1, and thus description thereof willbe omitted. In addition, control of the system in FIG. 8 performs anoperation flow similar to that illustrated in FIG. 2. A second imagingunit 806 functions as a second imaging unit that acquires an image of atarget object.

In addition, the internal configuration of the image processing deviceand specifications of an operation flowchart generation screen, a methodfor registering a search pattern, and the like have configurationssimilar to those illustrated in FIGS. 3 to 5. It is assumed that variouskinds of setting information set in the production system illustrated inFIG. 1 and search information such as the installation supportinformation are saved through connection to an external storage unit andthe like via interfaces 303.

In this state, if physical arrangement, features, and the like of theimage processing device 801 to a work piece 810 in the replicated systemillustrated in FIG. 8 are reproduced with features and accuracyequivalent to those of the production system illustrated in FIG. 1, aprocess similar to the flowchart 403 illustrated in FIG. 4 can beperformed in the image processing of the image processing device 801.Furthermore, the flow illustrated in FIG. 2 including the imageprocessing flow can be also performed in this way, and thus a productionsystem (conveying system) having an operation capability equivalent tothat illustrated in FIG. 1 can be easily replicated (additionallyinstalled).

However, actually, even if arrangement and device production areperformed in accordance with the plan of the system when the productionsystem (the conveying system) is replicated, there may be assembly errorin each assembly process, or there may be an individual difference foreach device or each member. For this reason, such error is accumulatedin the entire system, and it is extremely difficult for a replicatedsystem to reproduce a similar system operation immediately after theassembly. Thus, after assembly of the system, many adjustment operationsand operation checking operations accompanying trial and error forreproducing a target system operation are necessary for each device andeach member.

FIG. 9 is a diagram illustrating a search pattern registration screen ofthe replicated system illustrated in FIG. 8. Constituent elements 502 to513 of the screen are the same as the constituent elements 502 to 512illustrated in FIG. 5, and thus description thereof will be omitted. Inaddition, various kinds of setting data of the image processing deviceare reproduced by reading processing parameters that have been set andsaved in each processing block of the image processing flowchart 403that has been saved at the time of initial system setting.

In addition, the setting of the search pattern setting area 504 andparameter settings of 505 to 510 relating to the measurement of aposition and a phase, in other words, measurement feature informationand the like for measuring a position and a posture of a target objectare obtained by reading data saved as search information.

Then, the image processing device 801 measures a position and a postureof the work piece described above by searching for the measurementfeature information described above from an image obtained by a camera.In addition, by performing a predetermined process such as a process ofgripping the target object described above using the robot 804 describedabove on the basis of the position and the posture of the target objectdescribed above as measurement results and conveying the target objectto a predetermined position of the conveying device 805 or the like or aprocess of performing a predetermined processing, a desired product ismanufactured.

However, at this time, in the camera 806 in the replicated system, forexample, there is a high possibility of a relative positional relationbetween the camera 806 and the work piece 810 or the tray 809 in whichthe work piece 810 is installed and the conveying device 803 not beingphysically reproduced with high accuracy. For this reason, asillustrated in FIG. 9, positions of the search pattern setting area 504and the work piece 810 do not coincide with each other. Thus, in thestate illustrated in FIG. 9, appropriate setting cannot be performed,and even when the pattern search (the measurement of a position and aphase) and image processing and the system as represented in the imageprocessing flowchart 403 are performed, a desired operation cannot bereproduced.

In such a case, a system installation staff needs to perform adjustmentby adjusting the position and the posture of the camera 806 using anadjustment stage 808 and the like of the camera or adjusting thearrangement and the like of a stand 807, the loading device 803, and thelike in some cases such that desired image measurement can be performed.In addition, also when adjustment is performed, it is necessary torepeat checking of a captured image obtained by the camera, checking ofthe operation of the image processing device, checking of the operationof the system, and the like many times while adjusting a relativepositional relation between the camera and various peripheral devices(the robot, the conveying device, the workbench, and the like). For thisreason, extensive efforts and time are required for securing aprocessing speed, accuracy, and reliability that are requested.

In contrast to this, in this embodiment, high efficiency of aninstallation operation of such an imaging device can be realized.

FIG. 10 is a diagram illustrating a camera installation support screenaccording to an embodiment, and FIG. 11 is a flowchart illustrating acamera installation support process according to an embodiment.

Hereinafter, an operation of a program for supporting adjustment of thecamera installation position according to this embodiment will bedescribed with reference to FIGS. 10 and 11.

A screen 1001 for supporting installation of a camera is illustrated,and a button for starting a camera installation support process programmay be provided on a setting screen 501 illustrated in FIG. 9, and thecamera installation support process program may be configured to bestarted if the setting of image processing does not go well or the like.In addition, a button for starting a camera installation support processprogram may be provided on a screen for generating a flowchart of imageprocessing like the flowchart area 402 illustrated in FIG. 4, andadjustment of the installation position of the camera and checking ofthe installation state may be performed before the setting of each imageprocessing in FIG. 4.

A button 1002 is used for selecting an input candidate of an image thatis an installation support target and is configured as a camera imagehere, and a stored image that has been imaged by a camera in advance orthe like may be read.

A button 1003 is used for designating a camera that is an installationsupport target, and here a camera 806 is selected. In an imageprocessing system in which a plurality of cameras are connected, aplurality of cameras may be configured to be selectable, and switchingbetween information sources (the information saved in advance in FIG. 7)for supporting installation of a camera may be performed in accordancewith the selected camera.

A display area 1004 is used for displaying an input image of the camera806 and information for supporting installation of a camera in asuperimposing manner.

A check box 1005 indicates whether or not a camera installation supportprocess is performed, and when the check box is checked, the camerainstallation support process is performed. For example, if an arithmeticoperation load for image processing for camera installation support ishigh, and the image processing causes an obstacle in other imageprocessing and operations, this check bock may be turned on only whenthe camera installation adjustment is performed. Details of the camerainstallation support process will be described in detail with referenceto a flowchart illustrated in FIG. 11.

A button 1006 is for selecting a use feature ID used for the camerainstallation support process. In other words, the button 1006 functionsas a setting unit that sets search information on the basis of an imageand can set predetermined feature information as search information onthe basis of results of evaluation (priority levels) performed by theevaluation unit.

Here, as described in the feature information 702 illustrated in FIG. 7and the like, the use feature ID is an ID number associated with featureinformation to which a specific evaluation value is assigned.

A result 1007 is a result of a search for installation support featureinformation corresponding to the use feature ID1 on an input image andis displayed with being superimposed in the input image.

The result 1008 of the search of the installation support featureinformation on the input image is represented as a numerical value orthe like. In this embodiment, as illustrated in the search method 507illustrated in FIG. 5, shape feature information is used as imageprocessing for measuring a position and a phase, and thus shape featuresare selected as installation support feature information for supportinginstallation of a camera in accordance therewith. However, as theinstallation support feature information, local shade information or thelike may be used, and, if the camera uses an imaging element, a stereocamera or the like that is able to obtain a three-dimensional shape,three-dimensional shape information may be used.

In addition, as a result 1008 of performing the image search using theshape feature information, a search score (similarity, a correlationvalue), an enlargement/reduction magnification at the time of closestcoincidence, a horizontal position, a vertical position, a rotationvalue in each axial direction, and the like are displayed. A useradjusts the position of the camera while referring to this numericalvalue. Generally, it is preferable to perform adjustment such that adegree of coincidence with the feature information for camerainstallation support that has been saved in advance becomes a maximum,in other words, the search score is the highest as possible, and theenlargement/reduction magnification is near 1.0 as possible, and theamount of deviation of each axial direction is near 0.

In addition, in this embodiment, although adjustment can be performedwhile referring to the search score, the magnification, and the amountof deviation in each axial direction, at least one of these three piecesof information may be displayed, and adjustment may be performed whilereferring to it.

Guide information 1009 defines axes of an input image with thehorizontal direction set as the X axis, the vertical direction as the Yaxis, a depth direction of an image forming surface set as the Z axis.However, the axial directions may be set in accordance with situationsof the used camera and the image processing library.

Visual support information 1010 and 1011 support the installationposition of the camera. For example, directions of deviations in thedirections of the X axis and the Y axis, a direction of a deviation of arotation direction of each axial center, and the amounts of suchdeviations may be displayed like the visual support information 1010using directions of arrows, lengths of arrows, and the like, or theposition of the feature information of the original image that is atarget may be displayed in a superimposing manner like the visualsupport information 1011. In accordance with this, a user can visuallyobtain support information used for adjusting installation of thecamera. In this way, by displaying at least one of the adjustmentdirection and the amount of adjustment of the imaging unit on the basisof a result of the search, the adjustment process can be shortened.

A check box 1012 is used for executing an installation automaticadjustment program that automatically performs installation of a camera.By checking the check box 1012, the automatic adjustment function isturned on. This function, for example, can be used if the camera 806 orthe adjustment stage 808 of the camera includes an adjustment mechanism(the adjustment unit) of pan, tilt, position adjustment, zoom, and thelike. In addition, the adjustment unit may be configured to be able toadjust at least one of pan, tilt and zoom.

The automatic adjustment is performed such that various numerical valuesas represented in 1008 become appropriate while operating such anadjustment mechanism. In addition, the image processing system may beconfigured to automatically detect whether the adjustment function (theadjustment mechanism) as described above is present in the camera 806 orthe adjustment stage 808 of the camera and display the check box 1012 tobe checkable if it is determined that the adjustment function ispresent. On the other hand, if it is determined that there is noadjustment function, the check box 1012 may be displayed in gray or maynot be displayed such that checking is not able to be performed.

When adjustment is performed, a direction in which at least one ofnumerical values of the search score, the magnification, the amount ofdeviation in each axial direction and the like of 1008 becomes apredetermined value or less may be configured to be found by scanningthe entire movable range of the adjustment mechanism. Alternatively, adirection in which a weighted sum value of such numerical values becomesa value set in advance or less may be configured to be found.Alternatively, proportional control, feedback control of a PID, or thelike may be performed using an amount of adjustment of the adjustmentmechanism, various numerical values of 1008 changing accompanying suchadjustment, and the like.

A target value 1015 of error between feature information set at the timeof performing camera installation support and a search result isillustrated and, for example, here a success (completion of adjustment)is determined if the amount of position deviation in each axialdirection is within 5 pixels.

An OK button 1013 is used for ending the camera installation adjustmentsupport and ending this screen 1001 if preferable camera adjustment wasable to be performed while referring to the displays 1015.

In the case of automatic adjustment, the button 1013 may be configuredto be automatically turned on when the error converges to a target value1015. In the case of manual adjustment, a user may check that the errorconverges within the target value 1015 and end the adjustment by turningon the button 1013. When the adjustment ends, the use feature ID at thattime, the camera image, the information about the used camera, variousnumerical value information (the installation support information) 1008,and the like may be recorded together.

A cancel button 1014 is used for ending the camera installationadjustment support and ending this screen 1001 if preferable cameraadjustment cannot be performed even by using the feature informationprepared in advance, the search condition, and the like.

Next, as described above, FIG. 11 is a flowchart illustrating a camerainstallation support process according to an embodiment, and the processof each step illustrated in FIG. 11 is performed by a computer withinthe image processing system executing a computer program stored in thememory.

As illustrated in FIG. 11, a camera installation support process startsin Step S1101. A start timing is a timing at which the check box 1005 ofthe camera installation support process is clicked in the exampleillustrated in FIG. 10. However, other than that, a timing at which thesupport screen 1001 opens or the like may be set as the start timing.

Feature information used for a search is read in Step S1102. Since ID1is selected by the button 1006 as the use feature ID in FIG. 10, andthus as illustrated in 702 of FIG. 7, feature information and the likeof ID1 having the largest size are read as installation support featureinformation. In other words, Step S1102 functions as an informationacquiring unit (an information acquisition step) that acquiresinstallation support feature information for supporting installation ofthe imaging unit.

An image that is a processing target is input in Step S1103. In otherwords, an image that is the processing target is input from the camera806 or a storage device.

Step S1104 is a process of extracting features of the input image, andfeature information for a search process is extracted from the imageinput from the camera or the like. Here, since “shape feature” isselected as the search method, the shape feature information isextracted using luminance, gradient information, and the like of theimage.

Step S1105 is a feature information search process, and a search for aposition, a phase, an enlargement/reduction magnification, and the likeof feature information having a strong correlation coefficient with theinstallation support feature information read in Step S1102 is performedon the basis of the feature information extracted in Step S1104. Inother words, Step S1105 functions as search unit (a search step) thatsearches for the installation support feature information acquired bythe information acquiring unit from the image acquired from the imagingunit. When the search is performed, similar to a time when the searchpattern setting area 504 is registered on the setting screen 501 formeasuring a position and a phase illustrated in FIG. 5, a parametersetting of the button 505 to the perspective distortion range 510relating to measurement of a position and a phase may be used.

Alternatively, a search for a pattern may be performed using a settingscreen similar to 501 or a setting file. In addition, as describedabove, the search unit may perform a search using at least one searchmethod among “shape feature”, “normalized correlation”, “matching”, andthe like on the basis of the installation support feature information.

In this way, even if a deviation is present in the input image, a workpiece, or measurement environments, a search for feature information canbe robustly performed. In other words, if there is a slight differencebetween the initial system and the environment, feature information isdetected, and a numerical value and support information for correctingthe position of the camera can be output.

Step S1106 is a result output process.

The search result acquired in Step S1105 is output and displayed asnumerical values like the installation support information 1008illustrated in FIG. 10. At this time, Step S1105 functions as a displayunit that displays a search result acquired by the search unit andfunctions as an installation supporting unit (an installation supportstep) that supports the installation of the imaging unit described aboveon the basis of the search result acquired by the search unit.

In Step S1107, the search result acquired in Step S1106 is evaluated. Ifthe result is preferable (for example, the installation supportinformation 1008 is within the range of the target value 1015), Yes isdetermined, and the camera installation support process ends. On theother hand, if the evaluation is not preferable, No is determined, andthe process proceeds to Step S1109.

In Step S1109, it is checked whether the check box 1005 of the camerainstallation support process is turned on. If the checkbox is turned on,Yes is determined, and the process proceeds to Step S1110. On the otherhand, if the check box is turned off, No is determined, the processproceeds to Step S1108, and the processing flow ends.

In Step S1110, it is detected whether an adjustment function for beingable to adjust the direction, the position, and the zoom of the camerais present in the camera 806 or the adjustment stage 808 of the camera,and if the adjustment function described above is present, it isdetermined whether checking of the check box 1012 of installationautomatic adjustment of the camera is turned on.

Then, if Yes is determined in Step S1110, the process proceeds to StepS1111 of camera position automatic adjusting step. On the other hand, ifoff is determined, No is determined and, the process proceeds to manualposition adjustment step S1112.

In Step S1111, the camera position automatic adjustment is performed. Ifthe adjustment function (an adjustment mechanism) that adjusts at leastone of pan, tilt, and zoom of the camera is present in the camera 806 orthe adjustment stage 808 of the camera, the amount of adjustment isinstructed.

As described above, if the entire movable range of the adjustmentmechanism is scanned, for example, evaluation values are acquired whilean instruction value is changed using an appropriate interval value, andthe adjustment may end at a time point at which the evaluation valuebecomes a predetermined value or less. Alternatively, adjustment may becontrolled such that it continues until a best evaluation value isacquired. In other words, Step S1111 functions as an automatic adjustingunit that performs adjustment of a position or a posture of the imagingunit on the basis of the search result and adjusts at least one of thepan, the tilt, and the zoom.

If feedback control or the like is used in Step S1111, a nextinstruction value may be determined on the basis of a relation betweenthe previous evaluation value and the instruction value, and theadjustment may end at a time point at which the evaluation value becomesa predetermined value or less, or at a time point at which an evaluationvalue converges within a predetermined width. In Step S1111, when theautomatic adjustment instruction is performed, the process returns tothe input of an image in Step S1103.

In Step S1112, the position of the camera is manually adjusted. Forexample, the adjustment is performed if an adjustment mechanism of thecamera or an adjustment mechanism of the stage is a manual adjustmentmechanism, if adjustment to outside the movable range of the automaticadjustment mechanism is necessary, or the like.

In addition, in the flowchart illustrated in FIG. 11, after theadjustment of the camera (Step S1111 or Step S1112), the image inputstep S1103 is performed. However, instead of proceeding to Step S1103,it may be determined whether the evaluation value has been enhanced. Ifthere is no enhancement or the like, the use feature ID used for asearch may be switched to another ID, and the read step S1102 may beperformed. In such a case, it is preferable that the ID number bechanged such that the size of the feature information is decreased.

In this way, the installation position of the camera can be adjustedwhile trying feature information switching supporting installation of aplurality of cameras, and thus more preferable installation of thecamera can be performed.

FIG. 13 is an image measurement result checking screen. In FIGS. 5, 9,and 12, the same reference signs represent the same components, and thusdescription thereof will be omitted.

In order to perform image processing similar to that of the imageprocessing device 801 of the first system, an image processing device801 of each of the second and subsequent (replicated) systems calls theflowchart area 402 illustrated in FIG. 4. Then, by clicking Step S408that is a position/phase measurement processing unit inside theflowchart area 402, a setting screen 501 illustrated in FIG. 13 can beopened. In addition, on the camera installation support screen 1001illustrated in FIG. 10, setting may be performed such that an automatictransition to the next step is performed at a time point at which aninstallation target value 1015 is satisfied.

In FIG. 13, a measurement result of a position and a phase of a workpiece at the time of clicking a test execution button 511 is displayedin the image display area 503. In other words, a search is performedusing the search pattern setting area 504 set to the image processingdevice 101 of the initial (first) system, similar feature information isdetected from the inside of the captured image acquired by the camera806, and a result thereof is displayed.

1301 to 1306 highlight and display work pieces from which shape featuressimilar to the search pattern have been detected, and, similar to FIG.12, IDs are assigned in order of highest to lowest search score(similarity, correlation). 1303 has the highest score according to theinfluence of of a work piece being disposed in a tray 809 on whichphysical position restriction is weak, an individual difference, and thelike, and thus ID1 is assigned thereto. According to detectioninformation 1309 at this time, it can be understood that there is littleamount of rotation deviation in the X-axis direction to the Z-axisdirection (the camera and the tray and the work piece can face eachother), and the position is measured such that the place of the tray hashigh accuracy.

By performing such result checking display, it can be easily checkedthat an image measurement result that is equivalent to a result of imageprocessing of the initial (first) system is acquired in second andsubsequent camera installation operations in the second or subsequentsystems or the same system. Here, it is apparent that whether an outputnumerical value is in a preferable error range depends on theperformance and the like required by the system.

As above, in this embodiment, by using a camera and an image processingdevice adjusted in the initial (first) conveying system or productionsystem, a search pattern for measurement (measurement featureinformation) can be generated and recorded.

In addition, for supporting the second and subsequent equivalent systemsor installation of second and subsequent cameras in the same system,generation, storage, transmission, and the like of installation supportfeature information can be performed. Thus, an adjustment operation forinstalling a camera can be significantly reduced.

In addition, feature information is searched from the image, and thus,in the measurement of a position and a phase, search parameters such as505 to 510 can be adjusted and set. In other words, when the search isperformed on the basis of search information, the parameters for thesearch can be adjusted, and thus even if there are deviations in aninput image, a work piece and a measurement environment, featureinformation can be robustly detected. Thus, installation of cameras canbe flexibly supported.

In addition, as feature information, for example, various kinds ofinformation of the feature information such as a size of a featureshape, a magnitude of a shade deviation, and deviation information fromthe search pattern setting area 504 on an image for which a model ofwork pieces is generated can be analyzed. Thus, determination of a usercan be supported by evaluating feature information for performinginstallation of a camera in accordance with features of the process ofthe production system (for example, a background, conditions for workpieces to flow in, a stand and arrangement of devices, and the like) andoutputting evaluation values.

Then, in second and subsequent equivalent systems or in second andsubsequent camera installation operations in the same system, theinstallation support feature information described above and the settingdata are read or received. Then, by using such generated installationsupport feature information, the setting data, the captured imageacquired by a camera, and image processing, the reproducibility of theinstallation position and the posture of the camera are evaluated, andthe evaluation result can be output or an equivalent image processingresult can be output. In accordance with this, a user can performinstallation of a camera while checking the evaluation results (anumerical value, a support display, and the like), in other words, thereproducibility, and thus repetition of trial and error for adjustmentof the camera and checking of the operation of the system is notrequired, and the number of adjustment steps can be significantlyreduced.

In addition, the installation support feature information for performingcamera installation support can be selected again on the basis of theevaluation result of the feature information, and thus preferred featureinformation according to the configuration of the system, theenvironment of the image processing process, background information, andthe like can be easily selected. Furthermore, trial and error of theuser can be reduced.

In addition, if the camera or the adjustment stage of the cameraincludes an adjustment mechanism of pan, tilt, zoom, and the like, auser can try automatic adjustment before manual adjustment of theposition of the camera, the adjustment operation can be reduced.

Furthermore, if the camera or the adjustment stage of the cameraincludes an adjustment mechanism of pan, tilt, zoom, and the like, thearrangement of the camera, for example, can be also adjusted such thatvarious numerical values represented in 1008 illustrated in FIG. 10become preferable while manually operating such an adjustment mechanism.Thus, a user's manual adjustment process for the position and theposture of the camera can be shortened.

In addition, it can be easily checked that an image measurement resultthat is equivalent to a result of image processing of the initial(first) production system is acquired in second and subsequentreplicated production systems or in second and subsequent camerainstallation operations in the same system.

Furthermore, according to this embodiment, the production systemincluding a gripping unit that grips a target object on the basis of ameasurement result acquired by the measurement unit of the imageprocessing device described above can be efficiently replicated.

In addition, a product manufacturing method including a gripping step ofgripping a target object on the basis of a result of the measurementacquired by the measurement unit of the image processing device and aprocessing step of performing a predetermined process on the targetobject gripped in the gripping step can be efficiently replicated. Thus,an adjustment operation process in the second and subsequent productionsystems acquired by replicating the initial (first) production system orin second and subsequent camera installation operations of the sameproduction system can be significantly shortened, and the replicationefficiency of the production (product manufacturing) system can besignificantly improved.

While the present disclosure has been described with reference toexemplary embodiments, it is to be understood that the disclosure is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation toencompass all such modifications and equivalent structures andfunctions. In addition, as a part or the whole of the control accordingto this embodiment, a computer program realizing the function of theembodiment described above may be supplied to the image processingdevice through a network or various storage media. Then, a computer (ora CPU, an MPU, or the like) of the image processing device may beconfigured to read and execute the program. In such a case, the programand the storage medium storing the program configure the presentdisclosure.

This application claims the benefit of Japanese Patent Application No.2021-020421 filed on Feb. 12, 2021, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing device comprising at leastone processor configured to function as: an image acquiring unitconfigured to acquire an image; a feature information extracting unitconfigured to extract feature information from the image; and a firstgeneration unit configured to generate installation support featureinformation used for supporting installation of the imaging unit on thebasis of the feature information extracted by the feature informationextracting unit.
 2. The image processing device according to claim 1,further comprising an evaluation unit configured to evaluate the featureinformation, wherein the first generation unit generates theinstallation support feature information corresponding to the evaluationperformed by the evaluation unit.
 3. The image processing deviceaccording to claim 2, wherein the first generation unit assigns apriority level to the feature information in correspondence with theevaluation.
 4. The image processing device according to claim 1, furthercomprising a second generation unit configured to generate measurementfeature information for measuring a position and a posture of a targetobject on the basis of the feature information extracted by the featureinformation extracting unit.
 5. The image processing device according toclaim 1, wherein the feature information used for generating theinstallation support feature information is extracted from a first areaof the image, and the feature information used for generating themeasurement feature information is extracted from a second areadifferent from the first area.
 6. The image processing device accordingto claim 5, wherein the first area is larger than the second area.
 7. Animage processing device comprising at least one processor configured tofunction as: an information acquiring unit configured to acquireinstallation support feature information used for supportinginstallation of an imaging unit; a search unit configured to search forthe installation support feature information acquired by the informationacquiring unit from an image acquired by the imaging unit; and aninstallation supporting unit configured to support the installation ofthe imaging unit on the basis of a result of the search performed by thesearch unit.
 8. The image processing device according to claim 7,wherein the search unit is able to adjust parameters used for thesearch.
 9. The image processing device according to claim 7, wherein theinstallation supporting unit includes a display unit configured todisplay the result of the search performed by the search unit.
 10. Theimage processing device according to claim 9, wherein the display unitdisplays at least one of an adjustment direction and an amount ofadjustment of the imaging unit on the basis of the result of the search.11. The image processing device according to claim 9, wherein theinstallation supporting unit includes an automatic adjustment unitconfigured to adjust a position or a posture of the imaging unit on thebasis of the result of the search.
 12. The image processing deviceaccording to claim 11, wherein the automatic adjustment unit adjusts atleast one of pan, tilt, and zoom.
 13. The image processing deviceaccording to claim 7, wherein the search unit performs the search usingat least one search method among shape feature, normalized correlation,and matching on the basis of the installation support featureinformation.
 14. The image processing device according to claim 7,wherein the search unit is able to select a search method on the basisof a type of the imaging unit.
 15. The image processing device accordingto claim 7, further comprising a measurement unit configured to acquiremeasurement feature information used for measuring a position and aposture of a target object, search for the measurement featureinformation from the image acquired by the imaging unit, and measure theposition and the posture of the target object.
 16. A production systemcomprising at least one processor configured to function as: aninformation acquiring unit configured to acquire installation supportfeature information used for supporting installation of an imaging unit;a search unit configured to search for the installation support featureinformation acquired by the information acquiring unit from an imageacquired by the imaging unit; an installation supporting unit configuredto support the installation of the imaging unit on the basis of a resultof the search performed by the search unit; a measurement unitconfigured to acquire measurement feature information used for measuringa position and a posture of a target object and measure the position andthe posture of the target object by searching for the measurementfeature information from the image acquired by the imaging unit; and agripping unit configured to grip the target object on the basis of aresult of the measurement performed by the measurement unit.
 17. Aproduct manufacturing method comprising: acquiring installation supportfeature information used for supporting installation of an imaging unit;searching for the installation support feature information acquired inthe acquiring of installation support feature information from an imageacquired by the imaging unit; supporting the installation of the imagingunit on the basis of a result of the search acquired in the searchingfor the installation support feature information; acquiring measurementfeature information used for measuring a position and a posture of atarget object and measuring the position and the posture of the targetobject by searching for the measurement feature information from theimage acquired by the imaging unit; gripping the target object on thebasis of a result of the measurement acquired in the measuring of theposition and the posture; and performing predetermined processing on thetarget object gripped in the gripping of the target object.
 18. An imageprocessing method comprising: acquiring an image; extracting featureinformation from the image; and generating installation support featureinformation used for supporting installation of an imaging unit on thebasis of the extracted feature information.
 19. An image processingmethod comprising: acquiring installation support feature informationused for supporting installation of an imaging unit; searching for theinstallation support feature information acquired in the acquiring ofinstallation support feature information from an image acquired by theimaging unit; and supporting the installation of the imaging unit on thebasis of a result of the search acquired in the searching for theinstallation support feature information.
 20. A non-transitorycomputer-readable storage medium configured to store a computer programfor an image processing device to execute the following steps: acquiringan image; extracting feature information from the image; and generatinginstallation support feature information used for supportinginstallation of the imaging unit on the basis of the extracted featureinformation.